US (General) Exit Survey Analysis – 2024 - 2025 February¶

1. Introduction¶

Welcome! This report provides a qualitative, natural-language analysis of the US Exit Survey results form February 2024, until February 2025.

The verbatim analysis of the data was conducted primarily using machine learning methods to ensure consistency, accuracy, and fine-tuned control over the granularity and quality of the survey results.

2. Data Overview¶

This section contains:

A brief overview of the responses
A breakdown of the distribution
A sentiment analysis

2.1 Data Preview For February YTD US Exit Survey Comments (All Careers)¶

Month of Action Date	Bundle	Career	Country	Outcome Reason	Satisfaction Rating	Score Reason	Additional Comment
Jan-25	ultimate	US Pre K	United States	I'm not using the membership enough	2	Price	I attempted to cancel this in November and was surprised to see you charged me again in December and January. Perhaps co sister a cancelation process that doesn’t require 4 screens and lots of scrolling.
Sep-24	ultimate	slt	United States	Other	5	Variety of Materials	I used the trial to make a recommendation to others that I supervise. Very pleased and will recommend.
Apr-24	ultimate	US Grade 2	United States	I'm not using the membership enough	5	Ease of Website	I did not authorize this membership and would like my subscription cancelled and hopefully credited back to my card on file.
Apr-24	ultimate	ks2	United States	Other	3	Price	I didn't set a reminder to cancel, so this went through without me remembering to cancel a free subscription.
Sep-24	ultimate	esl	United States	I can't find the resources I need	4	Variety of Materials	I am strictly looking for materials for my Newcomer EL students. Most of the materials I find are too advanced or were not designed for ELs.
Apr-24	ultimate	US Grade 4	United States	I can't find the resources I need	4	Variety of Materials	I am looking for more resources for older students. Many of the materials that relate to my content are for younger students.
Oct-24	ultimate	US Pre K	United States	I can't afford it right now	6	Variety of Materials	I don't like it taken off my debit card. Every month my card gets locked because I have to pay for your membership and also charged a fee for you being international. I would rather not have the hassle of going to the bank every time my card is locked.
Sep-24	ultimate	US Kindergarten	United States	I can't afford it right now	7	Variety of Materials	I LOVE THIS WEBSITE! I’ve been able to teach many children through using this website. But unfortunately I can not afford to have subscriptions of any kind right now. Only able to pay for necessities
Jan-25	ultimate	sen	United States	I can't afford it right now	1	Price	I'm not using this app but kept on adding credit on my account. I'm not asking for refund but please stop getting credit from my account. Thank you
Dec-24	ultimate	US Grade 6	United States	Other	3	Customer Service	I HAVE TRIED TO CANCEL FOR THREE MONTHS I NEVER USE THIS! HOW DARE YOU KEEP CHARGING ME MONTH AFTER MONTH
Mar-24	ultimate	home ed	United States	I've changed careers	5	Variety of Materials	I needed shadow puppet templates. I used it once. Could I get a refund on the membership that just posted, after the free trial. TY
Nov-24	ultimate	US Grade 2	United States	Other	5	Variety of Materials	Please refund the payment you collected though I had seven more days BEFORE the subscription was due for renewal. Thank you.

2.2 Exit Reason Distribution¶

The chart below shows the primary exit reason — as per the Outcome Reason column — stacked by users' score reason, found in the Score Reason column.

3. Sentiment Analysis¶

This section includes:

A stacked sentiment analysis chart that allow you to see the distribution of sentiments across the different Careers
An overall sentiment distribution chart that allows you to easily see sentiment distribution across the whole data-set.

4. Topic Modeling¶

This section contains:

A stacked bar chart for visualizing the topics extracted from the Additional Comment column, grouped by Outcome Reason.
A line chart which shows us the evolution of topic frequencies over time.

The topics were extracted using BERTopic; an unsupervised topic modeling technique that leverages sentence embeddings to extract and group the semantic relevance of verbatim responses. BERTopic uncovers topics naturally as they emerge from the data, rather than looking for them from a predetermined set of labels. This allows us to capture new topics as they emerge organically.

4.1 Extracted Topics¶

Below you'll find a table displaying the topics that were extracted from this data-set.

Table Info:

Topic Nr.: This is a numerical representation of our topics.
Topic Name: An AI-generated label, produced by sending each topics' keywords, alongside a sample of responses each topic was modeled after.
- I recommend taking a look at the representation columns in order to gain a deeper understanding of the topics as the AI-generated labels can sometimes be a little too broad or narrow.
Representation: Top n representative keywords of a given topic
Secondary Representation: A re-ranked selection of keywords which focuses on creating a more diverse representation of the topic.
Representative Sentences: A sample of sentences from the Additional Comment column which comprise a given topic.

Topic Nr.	Topic Name	Representation	Secondary Representation	Representative Sentences
1	High School Math Resources	[middle school, special education, school, grade levels, high school, homeschool, homeschooling, highschool, grade level, 8th grade]	[school, teaching, middle school, level, elementary, homeschool, education, site, needs, kids]	[I am a middle school/ high school special education teacher, and I just don't see enough resources related to the populations I work with to justify keeping an account., Not middle school., Resources are great for elementary school, some for middle school, but they are too easy for high school.]
2	Difficult Membership Cancellation	[difficult cancel, tried cancel, actually cancel, hard cancel, cancel, canceling, cancel anyway, cancelling, trying cancel, need cancel]	[cancel, times, cancellation, canceling, trying cancel, tried cancel, cancelling, difficult cancel, thought cancelled, service]	[I have been trying to cancel for months., You make it very difficult to cancel., Very difficult to cancel membership.]
3	Refund Request Issues	[request refund, get refund, refund, please refund, like refund, refunded, money back, still charged, billed, charged]	[refund, email, card, charge, month, customer service, payment, bank, renewal, like refund]	[Now I'm being charged and shouldn't have been., I would like a refund., I would like a refund.]
4	Free Trial Confusion	[free trial, trial website, free, cancel free, trial, unlimited downloads, charged monthly, monthly charged, trial period, signed free]	[free trial, free, monthly, would, trial period, unlimited, signed, pay, fee, cancel]	[Thanks for the free trial., I signed up for a free trial membership for 30 days and I logged in today only to find out that this is not a free trial membership., I did the free trial and used it once.]
5	Free Resource Options	[great site, free trial, helpful, sources, site, free, helpful need, specifically, provided found, great]	[free, need, afford, love, free trial, access, searching, used, things, use enough]	[It's a great resource., I found some great options for resources during the free trial, but just cannot afford to pay for this subscription., I signed up for the free trial to use some resources.]
6	Budget Constraints and Value	[cost, justify cost, year price, cannot afford, budget, cant afford, expensive, expenses, save money, spending]	[need, price, product, 90, money, justify, need cut, use enough, save money, expenses]	[Finally, as much as $90/year is a bargain, I don't think I'll use $90 worth of materials in the year., I also am on a very tight budget as a single 40 year old and need to cut back on everything I can to better pay my bills., I do not use this enough to justify the cost.]
7	Gratitude and Appreciation	[thankyou, thank thanks, thanks thanks, thank thank, thanks thank, thanks, thank, thank thankyou, thankyou thanks, thank loved]	[thank thank, wishes, tight right, opportunity, best, support, customer service, quality, customer, wonderful quality]	[Thank you!, Thank you., Thank you!]
8	Selective Resource Downloading	[downloads, preview downloading, sites download, download, downloaded, want print, able download, like download, download everything, download use]	[print, downloading, preview, downloads, pages, google, computer, like download, printer, wanted print]	[I could download the pages but they would never open up so I could print., I do not like having to download everything, I just want to print resources in the moment., I signed up to download a 3d model for use, as the website said "download 3d model" - but when I went to download it, it instead opened the model in a viewer and only lets me take screenshots.]
9	Twinkl Resource Challenges	[twink, grades, teaching, high school, school, lessons, grade, curriculum, school year, education]	[school, love, time, teaching, using, curriculum, room, need, education, cancellation]	[I subscribed for one month to try to convince a colleague in whose room I'm working twice a week, just to show her the excellent quality of what Twinkl offers, far better resources that most other sites., L have loved Twinkl, but last school year I changed from High School Special Ed to on-level High School Chemistry and most of Twinkl's resources only go to 8th grade., twinkl was a lifesaver when I was teaching younger children, but when I made the move to upper grades, the curriculum had less wiggle room and I wasn't able to find as many things that were as applicable.]
10	Positive Website Feedback	[loved site, great site, great website, enjoy site, site great, love website, love site, website website, website, site]	[website, love site, user friendly, love website, used site, great site, website user, like site, absolutely love, using site]	[I LOVE THIS WEBSITE!, I love the resources and the site is user-friendly., I absolutely love this site!]
11	Future Return Plans	[back back, come back, coming back, likely back, back, go back, back next, may back, returning, return]	[come back, back future, may back, may come, back next, next year, changes back, career change, definitely come, hopefully able]	[I will be back!, I'll be back!, I will be back!]
12	Frustrating Business Practices	[money shady, website shady, business, shady, way business, predatory practice, online service, phone talk, ups way, marketing explaining]	[business, shady, emails, way business, worst experience, ashamed, waste time, predatory, response, trust]	[Awful way to do business!, That is a shady way to do business., Limiting downloads, locking out of accounts, long response time using chat, cutting people off who have waited on the phone to talk to someone and forcing them to leave a voicemail to wait 3 days for a response.]
13	Multilingual Educational Resources	[spanish site, level spanish, levels spanish, language spanish, spanish language, spanish, spanish user, language references, language business, languages]	[spanish, ell, download zip, grade, high school, reading, levels, teaching, math standards, spanish language]	[Please add more options for Spanish Language arts or the ease of selecting downloands in Spanish or other languages, I don’t want to download \n Zip files everyday just to get one or two worksheets and I don’t want to download zip files everyday to my computer or my phone., I am a world languages teacher looking for vocabulary/ grammar worksheets for levels A1/A2/B1 in French, Spanish, and Chinese, reading passages in the target language, and cultural unit supports.]
14	Education and Bible Materials	[religious based, catholic books, secular educational, secular, catholic, denominational bible, many catholic, heavily christian, non denominational, bible school]	[catholic, books bible, catholic books, needed family, non denominational, one kamala, parish, parish provides, party quality, heavily christian]	[The quality of the content was hit or miss and there is lots of heavily Christian content that is promoted as secular educational materials, I primarily teach non-denominational Bible study for 1st and 2nd graders and many materials were catholic in material., your "Catholic Books of the Bible" coloring sheet did not have all of the Catholic books of the bible.]
15	Free Planner Complaints	[free planner, planner available, planner getting, planner, wanted planner, planner given, planners, planner asked, planner never, planner told]	[planner, free planner, planner asked, planner mentioned, planner month, planners, planner kept, planner next, planner months, planner told]	[I was asked about a planner and since the planner i was getting funds thing from me no thank you delete delete deleted, I was told free perk for joining would be planner 3 months later no planner or tracking., I clicked to receive the Free Planner and a month has passed and have received no confirmation or communication regarding the free promotional planner]
16	Recommendations and Giveaways	[keep mind, future amazing, amazing reccomend, mind future, future, future often, everyone keep, best ever, best, amazing]	[mind future, people reason, pleased would, reason site, site others, site would, 10 would, others 10, would everyone, give away]	[Will keep in mind for future., I’ll keep you in mind for the future., I won one of the 100 give away Stanley's a year ago and it was the best ever!]
17	Farewell Messages	[god bless, bless, avenir gracias, bless good, le repondrai, repondrai dans, bless god, abonnement mais, gracias, aimais beaucoup]	[god bless, goodbye, sincerely donald, thank goodbye, financement maintemant, lenoir bye, le repondrai, gracias, goodbye thank, goodbye sincerely]	[God bless & good luck, Peut-être je le repondrai dans l'avenir., J'aimais beaucoup l'abonnement mais financement maintemant c'est difficile.]

4.2 General Topic Visualization¶

This section shows you a frequency bar chart of each topic, grouped by Outcome Reason.

On hover, this chart will show you the topic frequency for each stack, alongside the top 5 words from that topic's representation.

4.3 Dynamic Topic Visualization¶

This section shows us how topic frequencies evolve over time.